Differentiable Search Indices (DSIs) encode a corpus of documents in the parameters of a model and use the same model to map queries directly to relevant document identifiers. Despite the strong performance of DSI models, deploying them in situations where the corpus changes over time is computationally expensive because reindexing the corpus requires re-training the model. In this work, we introduce DSI++, a continual learning challenge for DSI to incrementally index new documents while being able to answer queries related to both previously and newly indexed documents. Across different model scales and document identifier representations, we show that continual indexing of new documents leads to considerable forgetting of previously indexed documents. We also hypothesize and verify that the model experiences forgetting events during training, leading to unstable learning. To mitigate these issues, we investigate two approaches. The first focuses on modifying the training dynamics. Flatter minima implicitly alleviate forgetting, so we optimize for flatter loss basins and show that the model stably memorizes more documents (+12\%). Next, we introduce a generative memory to sample pseudo-queries for documents and supplement them during continual indexing to prevent forgetting for the retrieval task. Extensive experiments on novel continual indexing benchmarks based on Natural Questions (NQ) and MS MARCO demonstrate that our proposed solution mitigates forgetting by a significant margin. Concretely, it improves the average Hits@10 by $+21.1\%$ over competitive baselines for NQ and requires $6$ times fewer model updates compared to re-training the DSI model for incrementally indexing five corpora in a sequence.

Large language models (LLMs) have shown impressive results across a variety of tasks while requiring little or no direct supervision. Further, there is mounting evidence that LLMs may have potential in information-seeking scenarios. We believe the ability of an LLM to attribute the text that it generates is likely to be crucial for both system developers and users in this setting. We propose and study Attributed QA as a key first step in the development of attributed LLMs. We develop a reproducable evaluation framework for the task, using human annotations as a gold standard and a correlated automatic metric that we show is suitable for development settings. We describe and benchmark a broad set of architectures for the task. Our contributions give some concrete answers to two key questions (How to measure attribution?, and How well do current state-of-the-art methods perform on attribution?), and give some hints as to how to address a third key question (How to build LLMs with attribution?).

变压器模型的缩放属性引起了很多兴趣。但是，在研究不同电感偏差和模型体系结构的缩放特性的效果的前提下，没有做太多事情。模型体系结构的规模不同吗？如果是这样，归纳偏置如何影响缩放行为？这如何影响上游（预训练）和下游（转移）？本文对十种不同模型体系结构的缩放行为进行了系统研究，例如变压器，交换机变压器，通用变压器，动态卷积，表演者以及最近提出的MLP混合物。通过广泛的实验，我们表明（1）架构在执行缩放时确实是一个重要的考虑因素，并且（2）最佳性能模型可以在不同的尺度上波动。我们认为，这项工作中概述的发现对当前在社区中评估模型架构的方式具有重要意义。

基于变压器的大语言模型（LLM）的最新进展已导致许多任务的性能改进。这些收益随着模型的大小而大幅增加，可能导致推理时间缓慢且昂贵的使用。但是，实际上，LLMS制造的一代人由不同的难度组成。尽管某些预测确实从模型的全部容量中受益，但其他延续更为微不足道，可以通过减少的计算来解决。在这项工作中，我们介绍了自信的自适应语言建模（平静），该框架用于动态分配每个输入和生成时间段的不同计算。提前退出解码涉及我们在这里解决的几个挑战，例如：（1）使用什么信心措施； （2）将序列级别的约束连接到局部人口退出决策； （3）由于以前的令牌中的早期退出而返回丢失的隐藏表示形式。通过对三个不同文本生成任务的理论分析和经验实验，我们证明了框架在减少计算的效果 - 潜在的速度最高为$ \ times 3 $ - 同时可维持高性能。

尽管最近的多任务学习和自然语言处理的转移学习成功（NLP），但很少有效地研究了在训练中缩放任务数量的效果。迈出了这一目标，介绍了Exmix（极端混合物）：跨越各个领域和任务家庭的大规模收集107个监督的NLP任务。使用EXMIX，我们研究了最大规模的多任务预培训的影响，并分析了普通任务家庭之间的共同培训转移。通过此分析，我们表明手动策划用于多任务预训练的理想任务，并不简单，而且多任务缩放可以自行改进模型。最后，我们提出了Ext5：使用自我监督跨度去噪和监督EXMIX的多任务目标预先训练的模型。通过广泛的实验，我们表明Ext5优于超级格，宝石，彩虹，封闭书QA任务的强大T5基线，以及Exmix之外的几个任务。 Ext5在预训练时也显着提高了样品效率。

我们为神经机翻译（NMT）提供了一个开源工具包。新工具包主要基于拱形变压器（Vaswani等，2017）以及下面详述的许多其他改进，以便创建一个独立的，易于使用，一致和全面的各个领域的机器翻译任务框架。它是为了支持双语和多语言翻译任务的工具，从构建各个语料库的模型开始推断新的预测或将模型打包给提供功能的JIT格式。

高级深度学习（DL）算法可以预测患者基于乳房成像报告和数据系统（BI-RAD）和密度标准的患者发育乳腺癌的风险。最近的研究表明，多视图分析的结合改善了整体乳房考试分类。在本文中，我们提出了一种新的多视图DL方法，用于乳房X线照片的Bi-RAD和密度评估。所提出的方法首先部署深度卷积网络，用于分别对每个视图进行特征提取。然后将提取的特征堆叠并馈入光梯度升压机（LightGBM）分类器中以预测Bi-RAD和密度分数。我们对内部乳房数据集和公共数据集数字数据库进行广泛的实验，用于筛选乳房X线摄影（DDSM）。实验结果表明，所提出的方法在两个基准数据集中突出了巨大的边距（内部数据集5％，DDSM数据集10％）优于两个基准分类方法。这些结果突出了组合多视图信息来改善乳腺癌风险预测性能的重要作用。

In this paper, we propose a novel technique, namely INVALIDATOR, to automatically assess the correctness of APR-generated patches via semantic and syntactic reasoning. INVALIDATOR reasons about program semantic via program invariants while it also captures program syntax via language semantic learned from large code corpus using the pre-trained language model. Given a buggy program and the developer-patched program, INVALIDATOR infers likely invariants on both programs. Then, INVALIDATOR determines that a APR-generated patch overfits if: (1) it violates correct specifications or (2) maintains errors behaviors of the original buggy program. In case our approach fails to determine an overfitting patch based on invariants, INVALIDATOR utilizes a trained model from labeled patches to assess patch correctness based on program syntax. The benefit of INVALIDATOR is three-fold. First, INVALIDATOR is able to leverage both semantic and syntactic reasoning to enhance its discriminant capability. Second, INVALIDATOR does not require new test cases to be generated but instead only relies on the current test suite and uses invariant inference to generalize the behaviors of a program. Third, INVALIDATOR is fully automated. We have conducted our experiments on a dataset of 885 patches generated on real-world programs in Defects4J. Experiment results show that INVALIDATOR correctly classified 79% overfitting patches, accounting for 23% more overfitting patches being detected by the best baseline. INVALIDATOR also substantially outperforms the best baselines by 14% and 19% in terms of Accuracy and F-Measure, respectively.

Modern deep neural networks have achieved superhuman performance in tasks from image classification to game play. Surprisingly, these various complex systems with massive amounts of parameters exhibit the same remarkable structural properties in their last-layer features and classifiers across canonical datasets. This phenomenon is known as "Neural Collapse," and it was discovered empirically by Papyan et al. \cite{Papyan20}. Recent papers have theoretically shown the global solutions to the training network problem under a simplified "unconstrained feature model" exhibiting this phenomenon. We take a step further and prove the Neural Collapse occurrence for deep linear network for the popular mean squared error (MSE) and cross entropy (CE) loss. Furthermore, we extend our research to imbalanced data for MSE loss and present the first geometric analysis for Neural Collapse under this setting.

We propose a new causal inference framework to learn causal effects from multiple, decentralized data sources in a federated setting. We introduce an adaptive transfer algorithm that learns the similarities among the data sources by utilizing Random Fourier Features to disentangle the loss function into multiple components, each of which is associated with a data source. The data sources may have different distributions; the causal effects are independently and systematically incorporated. The proposed method estimates the similarities among the sources through transfer coefficients, and hence requiring no prior information about the similarity measures. The heterogeneous causal effects can be estimated with no sharing of the raw training data among the sources, thus minimizing the risk of privacy leak. We also provide minimax lower bounds to assess the quality of the parameters learned from the disparate sources. The proposed method is empirically shown to outperform the baselines on decentralized data sources with dissimilar distributions.